In our daily life, both camera and digital camera encounter unsolvable problems in capturing a scene. For example, when photograph is taken in a tourist spot, a stranger may stand behind the target persons and also be captured into the photograph, or when the target persons finishes posing, a stranger walks through the camera lens and is also captured in the photograph. These are the problems in taking a picture. Furthermore, in famous tourist spots, the picture may not be taken again without getting in the line again. Subsequently editing the unwanted person out of the digital photograph is time consumptive and requires skill in image processing.
In previous related researches, texture synthesis and image inpainting construct the fundamentals of filling the lost region in image. Texture synthesis can be used to fill the large hole of input texture, while image inpainting can be used to repair the scratches of image. In computer vision, texture synthesis algorithms generate large similar texture from sample texture or fill the lost region of input texture called constrained texture synthesis. Image inpainting algorithms are used to repair the scratches, cracks and to remove texts from old photograph and paintings. Generally speaking, texture synthesis is applied to single texture and image inpainting is used in general image with multiple textures.
The conventional texture synthesis algorithm aims for the synthesis of a single texture. This is usually accomplished by comparing the similarity of the adjacent pixels and synthesizing the pixels with the highest similarity into the lacuna region which is left after removing the unwanted object. From the aspect of the synthesized pixel number each time, there are pixel-based and patch-based methods. L. Y. Wei and M. Levoy proposed a pixel-based method in reference: (1) “Fast texture synthesis using tree-structured vector quantization,” in Proc. ACM Conf Computer Graphics (SIGGRAPH), pp. 479-488, July, 2000. Another pixel-based algorithm in reference: (2) “Texture synthesis by nonparametric sampling,” in Proc. IEEE Int. Conf. Computer Vision, vol. 2, pp. 1033-1038, September, 1999 proposed by A. Efros and T. K. Leung is slow and fails in structural texture. L. Liang et al. proposed a fast patch-based method in reference: (3) “Real-time texture synthesis by patch-based sampling,” ACM Trans. on Graphics, vol. 20, pp. 127-150, 2001, but it is not suitable for the general image with many kinds of textures.
Considering the image inpainting algorithms, there are also pixel-based method, as proposed in reference: (4) “Missing data correction in still images and image sequences,” ACM Multimedia, December, 2002 by R. Bornard et al., and block-based method, such as proposed in reference: (5) “Region filling and object removal by examplar-based image inpainting,” IEEE Trans. Image Processing, vol. 13, September 2004 by A. Criminisi et al. The blocks for each pixel on the boundary of the lacuna region are used for comparison with the source region to obtain filled blocks with the highest similarity. The block-based method often results in block effect in the target region. The priority updating step is needed in each time of filling process. In addition, the extra color space transformation is also needed in previous conventional image inpainting algorithms.
Moreover, an algorithm which integrates texture synthesis and image inpainting is proposed, as in reference: (6) “Simultaneous structure and texture image inpainting,” IEEE Trans. Image Processing, vol. 12, no. 8, August, 2003 by M. Bertalmio et al., and inwardly extends the boundary of lacuna region by Partial Differential Equation (PDEs). However, the disadvantage of this method is the blurring in the target region.